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Who am I? 


: Infrastructure Engineer 

: Platform Team TechLead 
- 12+ years in development 
: 4 years in ManyChat 


: Specialization: high-loaded 
systems, performance 
engineering and fail-safe 
solutions 
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What is ManvChat 


1M+ 190+ 1B+ 140+ 


Businesses Countries use Conversations People are 
chose ManyChat powered in ManyChat 
ManyChat across the by ManyChat team 


for growth world every year 
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Usecases of 
events DB 


1 Messages 
Actions (external / 
internal) 


Historical state 
storage 


Unassigned 
Q Facebook 


HiDima, you attended our event yesterday, tell me 
how are you? 


| like it! 


| didn't like it :( 
We are glad that you liked everything © 


We suggest leaving you a phone number, we will 
add you to our list and you will be able to receive 
invitations to the most secret events 


Yes, ready! 


Not ready 


Enter your phone number 


Reply Note 


The last interaction with this user was more than 7 days ago. 


You can use Broadcasts to reach out to users like this. 


«> 


о 
Dima Taranov О 


(Z Subscribed (Unsubscribe) 
O Male 


@ Russian 
Contact 20:09 
Time: (UTC+00) 


2288442387948186 


All Channels History 


Pause Automation 
+ 30 minutes 


Automatically 


Contact Tags + Add Tag 
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The Trouble 


T окы 


PostoreSQL Database 
with events 


бАТВ of data 
Events are cold and 
rarely read 


10 ООО$ 
monthly price... 
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Our event 
databases lineup 


> 0.1% 0.6% 


1000 shards 1000 shards 


READ ONLY 


MANYCHAT 


1000 shards 


96% 


1OOO shards 


READ WRIIE 


06 


REMOVE 


USELESS DATA? 


MANYCHAT 


MANYCHAT 


REMOVE OLD A LITTLE 
DAIA OF FREE DIRIY AND 

ACCOUNTS? QUESTIONABLE 
WAY 


DO WE HAVE 
CHANGE ENOUGH ТІМЕ ТО 
THE DATABASE? GARE EAHERT Tab 
AND COLLECT 


BUGS? 
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Discovering 
a solution 


But, what do we want actually? 


DECREASE COSTS! 
T 
DECREASE ІНЕ DATA SIZE 
T 
CHANGE IHE DAIA STORAGE 
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Our targets 


1 


Parallel stateless 
component 


Fast access by scope 
(index similarity) 


Fast and stable 
Encryption 


Compression 


MANYCHAT 11 


6 Ability to rollback 
(this is an experiment) 


7 Low costs of storage 
З Low costs of support 


9  Repeatable case (that's 
not only a single database) 


10 Very low price 
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Change event 
databases lineup 


COLD STORAGE READ ONLY READ WRITE 
1. Last in a call chain 2. Empty resultset while a 3. When a donor is disabled, the 
(not required) donor is in migration cold storage begins to work 


automaticly 
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Discovering 


a solution 


=~ ON N ы 


Major slice of event 
count per subscriber 


Chunk 
size 


Upload / 
Download costs 


Compression / 
Decompression costs 


MANYCHAT 


Metrics for making a decision 


5 Encryption / 
Decryption costs 


Total price including 
development, storage 
and usage 
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Main slice migration 


8 000 000 


6 000 000 


4 000 000 


2 000 000 


10 50 100 200 500 400 500 
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Discovering 
a solution 


MANYCHAT 15 
The choice of metrics: 


7 >= ZU events 
to cold storage (70 - 2500) 


« 7O events for direct 
migration 


Brotli for | 
compression 


AES-256-CBC for encryption 


Оо + W N 


AWS 53 as a storage 
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Main Aspects of 
the Migration 
Process 


MANYCHAT 


1. Have migration data 


2 Step by step migration (chunk > 
compression > encryption > upload to s3 > 
insert link events to a database) 


3 Database should be really “cold” 


4 Remember about database capacity 


D Consistency control 


16 


HIGHLOAD2.0 > YEREVAN 2022 


account 


subscriber 


events 


70 


70 


Main Aspects of 
the Migration 
Process 


status 


created 


chunk created 


encrypted 


cleaned 


compressed 


events created 


MANYCHAT 


migration 


step data 


(some data": 


(some data": 


(some data": 


(some data": 


(some data": 


(some data": 


1j 


1j 


1j 


1j 


1j 


1j 


metrics 


(some metrics": 


(some metrics": 


(some metrics": 


(some metrics": 


(some metrics": 


(some metrics": 


1j 


1j 


1j 


1j 


1j 


1j 
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Main Aspects of Step by step 
the Migration 
Process 


[2022-11-28 18:56:16.00]: db: 62% compress: 3% encrypt: 4% upload: 27% links: 4% clean: 0.2% total time: 12.25 
[2022-11-28 18:56:16.00]: db: 57% compress: 3% encrypt: 4% upload: 32% links: 4% clean: 0.1% total time: 34.1s 
[2022-11-28 18:56:16.00]: db: 79% compress: 3% encrypt: 4% upload: 10% links: 4% clean: 0.2% total time: 13.15 
[2022-11-28 18:56:16.00]: db: 2% compress:3% encrypt:4% upload: 87% links: 4% clean: 0.2% total time: 17.65 

[2022-11-28 18:56:16.00]: db: 12% compress: 3% encrypt: 54% upload: 27% links: 4% clean: 0.2% total time: 54.95 
[2022-11-28 18:56:16.00]: db: 12% compress: 53% encrypt: 4% upload: 27% links: 4% clean: 0.2% total time: 51.25 
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Main Aspects of the Selecting rows 
Migration Process 


J 


100000 200000 500000 400000 
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Main Aspects of 
the Migration 
Process 


MANYCHAT 


Consistenoy control 


1. You can check everything... 


however with a longer 
process than migration 


2 Or, check it by numbers 


2 Live checking based on 
a sample 
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Main Aspects of 
the Migration 
Process 


MANYCHAT 21 


Switch shards and disable 
a database 


The easiest... and 
the most scary 


Switch shards 
by flag 


Shutdown 
a database 


Drop a database 


BU N P. 
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How does it work 


Link events 

DIRECT REVERSE 

d i 

"səKey": "5ЗКеу"': 

"event/4000/97/300897/4000 300897 9427.data. "event/4000/97/300897/4000 300897 9427.data. 
br.enc", br.enc", 

"first. event id": 3823, "last. event. 1а": 9427, 

"min, event timestamp": "2019-11-29 03:10:43.6", "min, event timestamp": "2019-11-29 O3:10:43.6", 
"max. event timestamp": "2020-05-23 14:35:37.8", "max. event timestamp": "2020-05-23 14:35:37.8", 
"size": 136 "size": 136 

) ) 


EVENT ID = 9427 EVENT ID = 5825 
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How does it work 


Engine 


DESC 


MANYCHAT 


I. 
ч! 
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How does it work 


Engine 


ASC 


P 
ч! 


24 


HIGHLOAD2.0 э YEREVAN 2022 MANYCHAT 25 


Post factum 
decisions 


Provider metrics 15 minutes * Sum т 1h 2h 12h 14 3d 1w Custom [ KI v |x 


Hot accounts 


Count 


3.00k 
2.50k 
2.00k 
1.50k 
1.00k 
= Ж к т 
ALIN АА VAENE 2 CS A ZA 
> ——, & SUA 2-7 NS wx > — с, ессе - - А. 


ai em 
23:00 02:00 05:00 08:00 11:00 14:00 17:00 20:00 


Ө coldstorage:provider:fetch events 8 coldstorage:provider:get stored events @ coldstorage:provider:store link event @ coldstorage:provider:fetch event 
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Post factum J Event 
decisions ЕУ 
2 Runtime 


consistency 
control 
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Design 
mistakes 


MANYCHAT 


1. S3 chunk address 
without db id 


Migration to a 
production database 


NO 


2 And the one little 
mistake 
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The one little 
mistake 


Difference between 
tests and reality 


Always look at 
metrics 


He-migration 
price 


But... we have the 
one big plus... 


RB чм ы 
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What do we 
have now 


1. Simple, stable 
and cheap storage 


2 Decreased annual 
costs from 427 200$ 
to8 544$ 


5 Total saved 
500 000%- 


NI о л 


МАМҮСНАТ 


Scalable and repeatable case 


Equal uptime comparing only 
with only PostgreSQL database 


Freedom from useless expertise 
Sharing and unpredictable point 
of failure 


Keeping fidelity to the principle 
“Do it simple” 
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Metrics 


DATA SIZE 
FIRSI REQUEST TIME 


STORAGE PRICE PER MONTH 


MANYCHAT 


BEFORE 1 DB IN CS 5 DB'S IN CS 


226TB 9.75TB 
60-82MS 60-82MS 
35 600$ 190$ 


54ТВ 
60-82М5 


712% 
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YES 


COMPRESS 
MORE 


What could be done but doesn’t make sense 


MANYCHAT 


190$ 


The major point — costs of development 
and support, we decreased costs 


51 


MERGE SMALL 
CHUNKS 

OF SEVERAL 

SUBSCRIBERS 


What could be done but doesn’t make sense 


NORMALIZE 
EQUAL EVENIS 
AND DECREASE 

EVENI COUNT 


what could be done but doesn't make sense 
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Thank 
vou! 


TELEGRAM: 
@ANTON D ZHUKOV 


TWITTER: 
@AN TONDZHUKOV 


MAIL: 
AJQMANYCHAT . COM 


give 
you've 


liked or what could be 


You can rate the talk and 
improved 


Leave your feedback! 
a feedback on what 


ttt tt 
tt + tt x 

TX PEEVE 
ТГЕҒҒҒҒҒҒЧ 
ТГЕҒҒҒҒҒҒЧ 
ГЕҒҒХҒҒЧ 

Lë ais d dh dh gh gh gt gy. 
ТГЕҒҒҒҒҒҒҒЧ 
ҒГҒҒАҒҒҒХҒҒЧ 
Läb dh ais d gh gh ais gt gt gy. 
Läb dh dh d d dh gh gt gt aia. 
ГЕҒҒХҒХҒҒҒҒҒЧ4 
Sete tt tt 
äh sis sis sis dt sach ain ais sia 
s ia dh d d d gh a. 


